Towards High Performance Disaggregated Flash Storage with Programmable NIC

Existing storage disaggregation solutions all require remote host memory as a proxy to access storage devices. This creates a single bottleneck between CPU root complex and PCIe switch and induces too much DRAM footprint. Moreover, those software solutions introduce high tail latency due to commands batching, system interrupt and context switch. This latency overhead is non-trivial for fast NVMe flash storages. We propose S-Direct to direct access remote NVMe flash storage using a programmable NIC. This new design bypasses remote CPU and doesn’t require host memory as a proxy, thus removing the single bottleneck. S-Direct enables peer-to-peer communication among flash devices which brings a new dimension of flexibility to create new storage access commands. Our preliminary results show that S-Direct achieves up to 8.2x tail latency reduction without throughput drop compared with previous approaches.